Lecture 8 slides: Bayesian inference

According to the frequentists theory, it is assumed that unknown parameter θ is some xed number or vector. Given parameter value θ, we observe sample data X from distribution f(·|θ). To estimate parameter θ, we introduce an estimator T (X) which is a statistic, i.e. function of the data. This statistic is a random variable, since X. That is, the randomness here comes from randomness of sampling. Di erent properties of estimation characterize this randomness. For example, the concept of unbiasness means that if we observe many samples from distribution f(·|θ), then the average of T (X) over the samples will be close to the true parameter value θ, i.e. Eθ[T (X)] = θ. The concept of consistency means that if we observe many samples from distribution f(·|θ), then the distribution of T (X) over these samples will be close in probability to the true parameter value θ, at least when we observe large samples, i.e. T (X) will be in a small neighborhood of θ in most observed samples. In contrast, Bayesian theory assumes that θ is some random variable and we are interested in the realization of this random variable. This realization, say, θ0, is thought to be the true parameter value. It is assumed that we know the distribution of θ, or at least its approximation. This distribution is called a prior. It usually comes from our subjective belief based on our past experience. Once θ0 is realized, we observe a random sample X = (X1, ..., Xn) from distribution f(·|θ0). Once we have data, the best thing we can do is to calculate conditional distribution of θ given X1, ..., Xn. This conditional distribution is called the posterior. The posterior is used to create an estimate for θ. Since we condition on the observations Xi, we treat them as given. The randomness in posterior is an uncertainty about θ. The Bayesian approach is often used in the learning theory.